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The purpose of the Colorado Evaluation Project was to 


field test the Common Status Measures at grades four and eleven in 
conjunction with a statewide assessment program based on 
objective-referenced testing instruments developed by the Colorado 
Department of Education for grades kindergarten, three, six, nine, 
and twelve. The evaluation was designed to determine educational 
needs and educational program performance in Colorado,. 
Multiple-matrix sampling techniques were used. .Results of several 
types of analyses are presented: (1) for each item and item pool, 
performance statistics for the total pupil sample and by 
sub-categories, including school program participation, urbanism, 
sex, ethnic background, second language spoken at home, and 
socioeconomic status; (2) a profile of those students in the iower 
20% on each subject tested for a given grade level; (3) performance 

_ statistics for each item and item pool for each school and district; 
(4) comparisons between national data for the Common Status Measures; 
(5) item quality, objective appropriateness and student performance 
for each objective-referenced item pool; (6) item intercorrelations 
for each subject at each grade level; (7) subject-by-subject 

. correlations within each grade level and Common Status Measures by 
subject correlations for adjacent grade levels; and (8) production of 
two tapes, one concerning occupational cognizance at grade 11 and one 
concerning language arts in grade 3.. (For related documents, see TM 
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The Colorado Evaluation Project was initiated to field test 
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the Common Status Measures (C$4) concurrently with Colorado's pilot 


_. program in assessment end evaluation. The project's successful com- 
pletion was possible only with the cooperation of many persons. 
Dr. Gene Glass of the dnivareiey ot Colorado advised the Colorado 
Department of Education (CDE) on item development for objective~ 
referenced items. The Belmont Group supplied interest end support 
throughout the entire project. United States Office of Education 
personnel were involved in many aspects of the project. Host iti- 
portantly, the educators and school children of Colorado gave the 
effort and time required to make the project worthwhila. 
The Purpose : 

The primary purpose of the Colorado fyaluation Project was . 


to determine Colorado's educational need, defined as "the discrep~ 


ancy between stated ebjectives and their achievement."" A secondary : 
purpose was to test assessment procedures thet may be replicated : 


or adapted by local school districts, by other states, and by 


federa] agencies. : 
Genesis : 
The interest of the Colerado Department of Education in precise : 
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assessment wes the proauct of two great current streams of concern 
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third great stream, The over-arching questions arising from these 
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; streams were these: 
tow well do Colorado stulents learn? 


fo 7 mete: a rr oe = ", 
What orc SUrOnGih. GANG wrasse 


If we want inprovesent, how do we got it? “ 
t 3 J 
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These questions are both scientific and political; scienti- 

fic in the realms of testing, measurement, sampling and computer 
analysis = political in the realms of allocation of iccal, federal 
and state resources, Science and politics constitute the streams 


of concern and thought mentioned sbove. 


duced a third stream and a nw term in the schools and legislative 
halle of our land: “Educetienal Accountability". Properly vndere 


stoud, accountability can lead us all to an ewarencss of common 


responsibility for the education of children in Colorado, Cur 
efforts and our resources ure expended for quality education. To 
this end Colorado educators, legisletors, and others are all ace 
countehic, 


VYorking Definitions co. 
ee at eon Sree 


Throughout the project, the investigators were guided by the 
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4 
: ; definition of “educational need! as "the discrepancy between objec 
i ; tives and their achievement", This definition, in its simplicity, 


entailed further explication as the project developed. These cx~ 
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plications follow, 


NObjective."! specifications for objectives shaped their daufi- 


nition. Colorado educators decided that each objective specify 


what a child should te able to do upon completion of a given year 
in school in terms of (1) the subject-content area (2) the ection 
or performance desirea, and (3) che measuring instrument or tech= 


nique. Further, cach goal was to be consistent with Coals fer 


res wad « a . 1, ae devea * atin Carrer - oe r s fitee, 8 
= Efucttiana in felorc. is, as adooted im blo? eng restated fn Tyo be 
ee ae ae mere een en te te ee ee i t a wad i 


the Coluradn State 30ard of Sducation, Finally, CvE personnel 


s 
ERIC 


; 


we 


ny MEI Mot 


ity 


tt pak RU uN PR 


-3~ 

specified that each objective be considered important by Colorado 
teachers. Accordingly, each objective was judged by a random sam~ . : 
ple of teachers in Caiorado as to their importénce or unimportance 


as year-end objectives, Level of importance was indicated by per 


cent of tcachers so responding. 


TABLE 1 
Number of Objectives Written 


by Grade Levels and Content Areas 


a eT tO LI EO LILLE LE OD, 


Level Health P.£&. Meth Lang, Arts Scicnce Music Total 
K $ 22 
3 28 12 10 an 12 S51 138 
6 27 15 17 V4 10 - 60 143 
9 7 26 16 22 hg 130 
wz 15 HM pe. a 
Totals 87 79 65 78 36 213 556 
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MAchievement."' Achievement was defined in terms of percentéce 


L 


of correct responses to test items relating to the objectives de- 
scribed above. ‘Items were developed and inspected for face validity 
with the besic questicn ~ "Does the item relly measure whet it 
appears to measure?! Roasing level, hidden meanings, ambiguities, 
and other feults were corrected as time permitted, Logical re- 
lationships of itcns to objectives were evaluated during item de- 
veiopaent and later dete analysis, The itens did not have to fix 
any indiviswrl's position with respect to norms, NOT predict his 


further success, nor correlate with other abilities, nor he easy 


wl} 


for some and hard for others, The items simply had to reveal whether 
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the individual could or could not perform a given task, 


"Anchor.'' The anchor concept developed rather late in the pro- 


ena 


vient ae ede 


vided an anchor for the Colorado Evaluation Project. By releting ‘ 
national scores ebtained previously on the CSA with scores obtained 
in the Colorado preject, new data could te compared with existing 


data to determine stability of the tests und testing procedures i 


used in Colorado, _ i 


"Educational need. " Derived from the above definitions, “edu- 


weer mere 


cations! need! was 3 construct with several facets, The need was 
felt by teachers who judged ecucat ional Objectives, by curriculum 
specialists who wrote the objectives, anc by pupils who were 


puzzled by some of the tesi iiv.is. Such discrepancies have been 


described in educational literature as the difference between the 


"ist! and Yought", between the “real! and "ideal", between "balance! 
and “imbalance! and many other theoretical constructs dealing with 


discrepancies. 
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Procedures for Statevi¢ 


Multiematrix sampling, hired test proctors, and computer 
analysis were basic procedures utilized fo: the Colorado Evaluation ~ : 
Project, ome 12,000 tests were administered in 209 schools in 3] . 


districts across the siate, Seroils are reported in the project's 


, ‘ interim report (June 15, 1$7G) and are summarized briefly below, 


Ul 
Sampling. Hultiesotrix sampling provides random sclection of : 


nen © 


both pupils and test fieus, In this way onpentys of every puoil 
er 


ate? en ‘ ities Bedi th agen 14 eae BA tek er ee } ' 
taking every ites gre tvoidoe watie needa th informstiln fs prosaic: + 


20.60 per day 
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Hiring proctors, Test proctors wore nired at 
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plus mileage to give the tests in the schools. This avoided dis~ 


ruption of ongoing teaching-learning school activities, 
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Analysis. Responses to test questions were key-punched ca 
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data cerds, A computer program of the Faectfic Educational Evalue 
ation System, Stanford, California, provided analyses comparing : 


pupil responses and pupil characteristics along with numerous 


edures as described in the following scetion 


° 
rt 
zs 
QO 
“tT 
re] 
o 
a) 
e 
w 
ct 
ie] 
te 
_ 
od 
“ 
9° 
o 


FInDISS 

The basic unit of information preduced were percentages of 
correct and incorrect responses ioe test item. This type of in- 
formation was tabulated to facilitate comparisons among various 
Student populeticns and among the items. Selected statistics! . 
operations indicated precision in these comparisons, 

Test scores arc discussed below on (i) the Convion Status 
Measures (2) the objective-referenced echicvement tests, and 


(3) comparison of results fre these two types of instruments, 


. The Coron Status Measures (CSM) 
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The Unitéd States Office of Education contracted in 1968 to 
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develop the CS#, U.S.0.£. Contract EC 0-9-09591 7-2! (O10). Under 

the U.S.0.E, @uspices, two T7~item pools were edkinistered in Calo- 

rado, ong item pool testing occupational cognizance, the other ; 

testing basic verbal status. foth 4th and Wih graders tried itens : 

; from both pools. Comparisons of the Colorado and national samples i 


are mod: in the fol] arms OF estimeted percertoqe of 
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correct responses of afl students teking all ftems. 
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TASLE 2 


Estimated Percent Correct 
for All Students on Atl items; 
Colorado Sample and Hetionel Sample on 
The Conmon Stetus Measures 


Seen eee 0 

= a So ——————— 
Comnon Status Colorazo latioaal ‘ 
Measure of: Sample Sample Differcnce 


Basic Verbal Grade & Grede It Grade Grace it Grade fi Grace 11 


Se ee Ne a NS et 


Status 7.3 66.) 67.6 59,0 6.7 7.1 


Occupation: | 
Cognizance 62.0 67.8 7.0 61.0 15.0 6.8 


—— 


it may be observed that Colorado scores are higher, The 


scores cppaar fairly consistent except et the fourth grade level 


in occupational cognizance. 


Occupetional connizance. Four elements comprised uccupa- 


tioral cognizance in the Comnon Status Heasures. These elements 


tested knowledge of a given dccupation regarding (1) education 
or trainitig needed, (2) work involved, (3) other occupations 
= related, and (4) field of work. Colorado and national scores 


were conwered on these four elements. 
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ABLE 3 a 

Comparisor. of Nationel with Colorado Scores* 
on Elements of Gecupational Knowieuge 
in Ath and 11th Grades 
Average 
7 ‘ Rational Colorado Difference 

. Elements tested: Ath With keh Vith hth 11th i 
Educarion or training needed 3656 50.65 i4 g ! 
i 
Mork involved 57 662 71. «65 14 3 
Occupetions related h§ 459 6; 66 16° #7 i 
. ; 
Field of work corresponding 52 65 69 69 17 h : 


-— Estimated porcontage correct for ¢ a1) pupils taking all items 


icales that Colorado's edge is larger in the 


Lee a etl pak ln de estat a acy RB 


hth than in the llth grade, Colorado }ith graders scores Nigher 


ooh 


than the national sample, indicated in the "difference colunn 


at the right. However, the differences achieved by the Colo- 
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rado ith eraders are considerably higher than those echieved 
by the Colorado lin graders, 

Perhcps the most useful information regarding neecs for 
occupational cognizence ctn be found in comparing student groups. 


Chi-square comparisons showed certain @roups to have wore than 
i c | 


. 


their share of low scorers. his information ves most useful 


in enswering questions posed at the outset of Lhe project re- 


gerding occupational cewsizence in Coloredo, 


Q; What are the distinguishing charactaristics of pupils dee 
ficient tn knowlecse of occupations? 
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scoring group; these students come froin families with low 
annual income. Similarly 4th graders from low socio-economic 


status families had more than their share of difficulty. ; 
Also, at both grade levels, American Indian, h 


’ ogrd an | 

A Spanish-surnacd American childre significantly had more 

difficulty than the total stv’en - -oulation. Eleventh 

graders who were from bili n 8 or who were girls 

wad wore than tneic snare vo. fhculby with Guest.ons on 

eccupations. A single low-scoring student may have several 

of these characteristics. 

: QO: Are there significant differences In tevel of cccupational 
cocnizance and basic verbal status with respect to urbenisn 
and location af school? 


Az: No significant difference in occupational cognizance Vas 
found relating to urbanisin of the school's district. Urban 

; students tended to cutscore cither susurbea or rural stu- 

: dents, but nol to a sionivicanc extent; such differences 
may have been dug to chance, District characteristics, 
stich as isolation or geographic location were not separ~ 

2 ated in this analysis, 


Q:; What ere the reletions among occupational cognizance and 
characteristics of school end progrei? 


As Somewhat surprisinsly, fourth gracers from schools with 
Title Uli progrems showed up with diaprogorcionacety bow 
scores On occupational cocnizance., Title I schools have 
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we 


Ve 
innovative programs not necessarily related to lou achieves 
ment, Low results in both fourth and cleventh grates also 
cane in from schools with both Title | and Title [ll pro- 


grams. 


Table 4 following gives the parcentace correct scored on 


itp tage Aen en LA 


. ow 


occupational questions. Colunn headings are explained in the 
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cxplanatory information above, 


Verbal status, Although Colorado pupils outscored the 


national sanple on verbel status, certain stucent poosuiations 
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within Coloredo missed more than their share ef those ites 
measuring yorbal status, Hore particularly, however, the cata 


from the Colorado Evaluation Project contein nswers to the 


questions posed at the outset of th 
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Groups of students with more than their share of difficulty 
on verbel items were these in the ith grade: Nero, Spanish- 


get Was etre Re oy 20 « +i 
surnamed Ania ricais Gu tO Soci Geeconomic status ci 


PUL S, 
t 
Distribution of cihnic minorities in Colorado is dascribed 
i the fecent CNS rece’ Coli ida Peon le one Colores £ cuce= 
ie Aliaos t iO of the i ieqro populetion 1S located in 
yvises beiween 10 and 1S per cent of that 


rural on 


pope ulation. 
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uburbs. 
did not show uo 
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See Table 5 following for suenary of above information, 
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Common Status Measures end f Needs Assassment 

It should be remembered that the Comson Status Neasures 
@re item-pcols not tests. That is, at the fourth-grade Icvel 
we have a collection of 72 questions thet sexple several aspects 
of pupils! basic verbal status, It follows then that we should 
look to perfermance en these items first when using the CSH's 
for the assessisent of educational needs 

If we continue to use Basic Verbal Stetus-Fourth Grace as 


ve could Jook at the performance of differen: 


our exemple 


groups of pupils (er schools, districts, states, regions, etc.,) 
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Items. Every question developed by the Colorado educators wes based 
on an explicit statement of an educational objective. As an example 


of how the results may be used, the PEES staff grouped the items by source- 


The performance of each category of pupil on every item is given on 


‘a at the Colorado Department 
eet a % ewe 3 °. . a ae . a, aig 6.5 * ae 

of Caucatron, educel ional gpeciarrsts in Goreraago, whe ure ClOSEr to 
the curricultia,will undoubtedly be eble to make more detailed interpre= 

tations. As one interprets the performanc? of pupils on an ites in 


relation to the corresponding chiuctive, several possibilities become 


evicent: 
1. tf the ite) appeals to be ndasy" that is a high percentage 
of pupils encer it correctly, 
(a) the objective me/ be successfully tauget 


(b) the neo LION May be poorly worded to “give-away” the 
sorrecty a15 
eiva mey have been inappropriate (oupils already 
winged the objectives im an earlier grade 
(d) the gues ion may not correspond to the objective. 
2, Wf the ile appaers to be difficult, 
(a) the objeccive may rot have been dealt with successtully 
by the curricufun, teaching natheds, or beth 


(b) the cuustion may be 30 poory worded thet pupils chose an 
incorrect response though they have, in reality, atteined 


the objective 
(c) the oijective my he inappropriate (that is, too advanced 
for the indicated grcde level 
(d) the question may mot correspend to the objective 


These possibilities were considered in the interpretations mace 


for each of the objoctive-referenced itens. 
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Two over-all compar isons yiclded information on reletions of (1) th 


CSN to ohjective referenced responses and (7) relations of student 
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Table 12 


Third - Grade Subject and Fourth « Grade 
Comon Status Measure Spearman Rank 
Order Correlation (Kkho)* 
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Bleventl - Grade Cowwin Status Measure 
ant Twelfth - Grade Subject Spearman 
Rank Order Correlation (kho)* 
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echievement of objectives to teacher's acceptance of objectives 


SH and objective-referenced correlations. Estimeted correlations 


(Spearman rhos based on school means) between the CSN and objective- 


tables should be interpreted with extreme caution. The rhos are 
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2 acnieverent item pacis 
difference among age groups in adjacent grades, and other statistical 
sources of imprecision. 


pupil achieveont. Kesponses to objectives 


were renorted for tivo operations: (1) the percentage of teccliers judg- 
ing the objective (9 be important and (2) the percentage of correct 
puoi] responses . These comporisens are shown on Craph #1 following 
Tables 12 and 13. 
Sunvsry 

The Colorado Evaluation Project encomoussed o wide range ef activi- 
ties ond required the narticipation of students, teachers, classr room 
teachers, school and district acainistrétors, personnel from tie 
Colorsce Department of Education and tne United States Office of Educa- 
tion, onde educations! consultants and sp2cialists, tech croup played 
important functions in this project. 

The purpose of the Colorado Evaluation Project § to ficld--test 
the Comvoa Stétus Moesures at crades four sad eleva. in congurctian 
with @e state-wide assessment program based on objective-referenced 


testing instruncnts dceveloged by the Coloreds Departs ent of Education 


for grades kindergarten, thrice, six, nine, and tuclve, The evaluetion 
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A distinquishina feeture of the Colorado Evaluation Project was 

its use of multiple-matrix sampling techniques to obtain reliable group 
data. Items were sampled from item pools for each subject at each grade 
love! Gnd randsuly éStidued to test forms, Siscriers, senaois, cinsses 
and pupils were also rencomly sampled, These techniques permit efficient 
ana economical educational progres cossessment. 

Results of severe! types of anelyses are presented: 

1. For epch ito end item pool, performance statistics were listad 
for the rotal pepil sompies and by virious sub-categories inctud- 
ing: sehecl procren pei diel verioa. wrbonism, sex, ethnic bock- 
ground, whether @ second langusese is spoken in the home, and 
socio-uconomic strtus. 

2. Statistical onarntions were verformed to chtain a profile of 
these stvdants wao perfors.ed in the lower 204 on cach subicct 
tested for & aiven grade level. 

3. Performance storictices were clso listed vor each iiem end iteum 
pool for each scheol and school district’ in the camnte, 

4, For the, Comnon Status Measures, cor oarisons betuoun nations! 
date cuthered during the Coterado Fyaivation Project were per- 
formed, 

5S. Item quality, chjective aporepriatenucs and student parforcance 
were discussed for cach puiseriverersrened 4 tem pool, 

G, For ecch subject at each grad: level, item intercorreiations 
wore obteinad, 

7. Subject by subtect correlations within ecch grece level and 
Covnon Status :csures by susicet correiations for ediacent 
arece levels vera aiso obtained. . 
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CONCLUS 10US 

H The confluence of scientific and political thought has produced a 
5 new era in determination of educational need. From the activities and 
i 3 
; fludings of the piict project in sesecemont end evaluetion these tenta- 
3 tive conclusions ware reached, . 

1, Eeucetional need is certasirated in relation to certain pupil 
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populations, This nes may be Tor cupfis to mgster certain objectives, 
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-o- the nowt ma, be for educators to form objectives more appropriate for 


the st Usey nogulation so identified. The neat appears to be covron in 


all subject arcas represented by t 
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i 2. Piscrepencics exist in dotorcda between whet teachers exrect 
: of their pupils and whet the pupils across the state can do, Each item 


of orch fort provides 2 fund of inforsatio, as to specific areas of 
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weekness. Interprat.tions of ciscrenancy inforantion will be darivad 
. o 

i both from the obieetive iiself end from pupil performance, The over- 
i 
i riding concern is to relieve the educational nacd indicated by the dis- 
i crepancy between ‘the cbjective and its achiove wnt. 
i . 
z 
i 3, Procedures of assessment oppear anequate to measure stateice 
z 
i objective achicvesent of common Icernings. Thet assessment system 
7 . ~ 
i would hove severa! innovative aspects : 
i o. Pupils, terchers, and seninistrators would lose minimal instruc- : 
t tional time to evaluation grccodures, 
z b, The system uae paintain devarmingtion of the criteria far z 
E é educutional suecess in the hends of Colorado ecucators. ‘ 
P 2 
: c. The test questions my be revised, in pert or as a whole, 4 
E improved, and updated, Following cach testing cycle. 7 
é ‘ ; 
E 5 
Z 2 
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d. The system is more economically efficient than traditional 
testing programs, 


e. The system insures that the evaluative data will he relevant 
to the planning needs of educational administrators. 


f. tnadcdition te giving edueatione! desiston-mekers tha Inferme- 
tion they recuire, the system provides a vchiclo for educators 
to refine and explicate their educational objactives. 
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Ca tie basis of the "Findings" and "conslusicas' discussed above, 


a 
: receerondstions are ofrcred For incressing accountability ih.oual essese~ 
went pad eveluecicu. ‘These recomiencations form a model for a continuing 


evelustion precréem in Celorede. 
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ews Blementary aust Pupil Characteristic Items 


Rame of Student: . es 


Scehoel: 


Name .of Woacher ‘ 


Dates 


ce ee OR AF 


Time of Day: _ 


Ky Kadi dst aida atonal welts seh HIMALA Ra iy 


So the proctor: Pacank ansiier the folowing queniions wiih the aid of 
tho sgtrdant and/or school personnel. he aid of school personne) in 
ansvering itea number four may be especially necded. Pupilts records 
ghana be checked to verify his responses. 


9. Indicate belev 72 this pupil is a member of any of the following 
reciel or national origin groups? 


) 
) 
) Oriented 


wersons of Cuban, Mexican, oF 
Puerto Rican descent) 
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